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FOREWORD 


Guidance  for  training  development  is  provided  in  US  Army  Training  and 
Doctrine  Command  (TRADOC)  Pamphlet  350-30,  Interservice  Procedures  for 
Instructional  Systems  Development  (ISD),  August  1975.  Some  of  the  training 
being  developed  and  conducted  does  not  follow  this  guidance,  however, 
because  many  of  the  training  developers  and  deliverers  have  not  been 
trained  in  the  development  of  a  training  process;  they  are  subject  matter 
experts  who  are  detailed  to  training  development.  Training  is  typically 
developed  under  a  concern  for  what  is  being  trained.  How  that  training  is 
developed,  conducted,  and  evaluated  is  usually  given  less  attention. 
Because  training  is  seldom  subjected  to  formal  evaluation,  ineffective 
training  may  go  uncorrected.  US  Army  Research  Institute  Fort  Knox  Field 
Unit  has  developed  a  system  for  formally  evaluating  the  training  process. 
Its  use  should  measurably  upgrade  training  development,  conduct,  and 
evaluation  in  the  Army. 

Training  Program  Evaluation  (TPE)  is  a  system  for  evaluating  the 
effectiveness  and  efficiency  of  a  training  program.  It  incorporates  the 
decision  rules,  data  collection  formats,  and  analysis  procedures  for 
evaluating  the  soundness  of  a  training  plan  as  expressed  in  lesson  plans 
and  training  materials,  evaluating  the  training  and  testing  process, 
identifying  training  and  non-training  causes  of  poor  soldier  performance  in 
training,  and  for  recommending  modifications  to  training  and  testing  that 
have  a  high  probablty  of  eliminating  poor  soldier  performance  in  training. 

This  job  aid,  one  of  four  in  the  system,  addresses  the  problem  of 
evaluating  lesson  plans  with  regard  to  the  adequacy  of  the  training  pre¬ 
scriptions  (descriptions  of  the  training  events/learning  experiences) 
contained  in  these  plans.  Guidance  is  provided  for  evaluating  training 
objectives,  training  process  plans  (plans  for  lectures,  demonstrations,  and 
practice  events),  and  testing  plans  and  Instruments.  This  job  aid  is 
useful  to  Army  personnel  in  centers,  schools,  and  operating  units  for 
evaluating  training  programs. 


EPH  ZETiDKER 
hnical  Director 
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BRIEF 


REQUIREMENT 

The  Army  does  not  have  a  standard  set  of  procedures  for  evaluating  the 
effectiveness  and  efficiency  of  training  programs.  A  need  for  such  standard¬ 
ized  formal  procedures  was  identified  by  the  Army  Training  Study  in  1978. 
Guidance  from  the  Army  Training  Study  specified  the  development  of  procedural 
guides  that  would  not  require  an  analyst  sophisticated  in  educational  tech¬ 
nology,  would  be  applicable  to  established  and  developing  weapon  systems,  and 
could  be  used  in  institutional,  transition,  and  sustainment  environments. 

PROCEDURE 

Training  Program  Evaluation  (TPE)  was  designed  as  a  system  for  evaluating 
the  effectiveness  and  efficiency  of  a  training  program.  Decision  rules,  data 
collection  formats,  and  analysis  procedures  for  evaluating  the  soundness  of  a 
training  plan  as  expressed  in  lesson  plans  and  training  materials,  evaluating 
the  training  and  testing  process,  identifying  training  and  non-training 
courses  of  poor  soldier  performance  in  training,  and  for  recommending  modifi¬ 
cations  to  training  and  testing  that  have  a  high  probability  of  eliminating 
poor  soldier  performance  in  training  were  incorporated  out  of  the  civilian 
and  military  literature,  industrial  practice,  and  the  experience  of  the  re¬ 
search  team. 

Observable  elements  of  the  training  process  were  specified.  These  ele¬ 
ments,  or  items,  included  such  things  as  whether  or  not  everyone  practiced  the 
training  task  to  standard,  if  the  training  aids  specified  by  the  lesson  plan 
were  used,  whether  or  not  tasks  were  demonstrated,  if  testing  was  contaminated 
by  unwarranted  prompting,  etc.  These  items  were  formatted  into  a  worksheet 
and  given  several  field  trials  with  typical  users.  The  major  field  trial  was 
conducted  in  conjunction  with  the  Ml  tank  OT-III. 

Following  the  series  of  field  trials,  lessons  learned  were  compiled  and 
the  items,  guidance,  and  suggested  worksheet  formats  were  finalized.  Suggested 
program  modifications  were  devised  to  correct  any  problems  found.  This  was 
done  separately  for  each  related  set  of  observable  items.  Program  modifica¬ 
tions  are  indexed,  therefore,  to  problems  observed  in  the  training  itself  or 
training  plans/materials.  Because  of  the  importance  of  training  objectives, 
practice,  and  feedback  to  training  effectiveness,  separate  sections  on  these 
topics  were  developed  using  training  literature,  experience,  and  lessons 
learned  as  guidance. 

There  are  four  job  aids  in  the  series;  one  for  modifying  ineffective  or 
inefficient  training  programs,  one  for  systematically  observing  training  and 
testing,  one  for  use  by  the  training  analyst  to  guide  him  or  her  through  the 
entire  process,  and  this  job  aid. 


FINDINGS 


The  TPE  system  has  been  used  in  several  operational  training  program 
evaluations  and  has  provided  training  process  data  not  heretofore  available. 
Users  have  found  these  data  useful  for  "fixing"  training  problems. 

UTILIZATION 

Preliminary  versions  of  the  TPE  materials  have  been  provided  to  the  Armor 
Center  and  School,  the  Armor  and  Engineer  Board,  and  the  Office  of  Armor  Force 
Management  and  Standardization  (OAFMS)  at  Fort  Knox,  the  Soldier  Support  Center 
at  Fort  Benjamin  Harrison,  the  Artillery  School  at  Fort  Sill,  the  Ordnance 
Center  at  Aberdeen  Proving  Ground,  the  Infantry  School  at  Fort  Benning,  the 
US  Army  Training  and  Doctrine  Command  (TRADOC)  Deputy  Chief  of  Staff  for 
Training,  TRADOC  Training  Development  Institute,  and  the  National  Defense 
Headquarters  at  Ottawa,  Canada.  Final  versions  have  been  requested.  In  addi¬ 
tion,  the  system  has  been  implemented,  all  or  in  part,  at  Fort  Hood  where  the 
TRADOC  Combined  Arms  Test  Activity,  TRADOC  Systems  Analysis  Activity,  the 
Armor  Center,  and  OAFMS  used  it  to  evaluate  the  New  Equipment  Training  for  the 
Ml  tank,  at  Fort  Knox  where  it  has  been  used  to  evaluate  the  Advanced  NCO 
course.  Ml  Tank  Basic  Armor  Training,  M60A3  Basic  Armor  Training,  and  at  Fort 
Bliss  where  it  has  been  used  as  a  baseline  for  a  system  for  evaluating  the  NET 
programs  accompanying  Air  Defense  Developing  weapon  systems. 
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SECTION  I 


PREFACE 

Each  job  in  the  Army  is  composed  of  a  set  of  duties.  These  duties  are,  in 
turn,  composed  of  performance  tasks.  Every  soldier  in  the  Army  is  respon¬ 
sible  for  being  proficient  on  a  set  of  tasks,  usually  compiled  into  a 
Soldiers  Manual.  To  insure  soldier  proficiency,  the  Army  trains  each 
soldier  on  each  task. 

The  US  Army  Training  and  Doctrine  Command  (TRADOC)  has  provided  guidance 
for  training  development  in  TRADOC  Pamphlet  350-30,  Interservice  Procedures 
for  Instructional  Systems  Development  (ISD),  August  1975.  Some  of  the 
training  being  developed  and  conducted  does  not  follow  this  guidance, 
however,  because  many  of  the  training  developers  and  deliverers  have  not 
been  trained  in  the  development  of  a  training  process;  they  are  subject 
matter  experts  who  are  detailed  to  training  development.  Training  is 
typically  developed  under  a  concern  for  what  is  being  trained.  How  that 
training  is  developed,  conducted,  and  evaluated  is  usually  given  less 
attention.  Because  training  is  seldom  subjected  to  formal  evaluation,  bad 
training  may  go  uncorrected.  This  job  aid  and  the  other  three  job  aids  in 
this  set  (listed  below)  provide  techniques  for  formally  evaluating  the 
training  process.  Their  use  should  measurably  upgrade  training  develop¬ 
ment,  conduct,  and  evaluation  in  the  Army. 

The  set  of  four  job  aids  addresses  the  evaluation  of  training  materials, 
training  observation,  how  to  correct  training  problems  discovered  during 
evaluation,  and  how  to  use  training  observation  and  test  data  to  analyze 
the  efficiency  and  effectiveness  of  training  programs.  These  four  job  aids 
are: 


Research  Product  81-15,  A  Job  Aid  for  the  Systematic  Evaluation  of 
Lesson  Plans,  which  will  help  you  evaluate  lesson  plans  for  adherence  to 
sound  training  development  principles. 

Research  Product  81-16,  A  Job  Aid  for  the  Structured  Observation  of 
Training,  which  will  help  you  and/or  the  training  observer  see  and  record 
useful  data  for  evaluating  training  programs. 

Research  Product  81-17,  A  Job  Aid  for  Modifying  Ineffective  or  Inef¬ 
ficient  Training  Programs,  which  will  help  you  decide  what  training  modifi¬ 
cations  should  be  made  to  correct  training  problems  discovered  during  a 
Training  Program  Evaluation  (TPE). 

Research  Product  81-18,  Guidelines  for  Conducting  a  Training  Program 
Evaluation  (TPE),  which  will  help  you  decide  if  there  were  performance 
deficiencies  on  in-course  tests,  the  likelihood  of  these  performance 
deficiencies  being  caused  by  training,  what  is  probably  wrong  with  the 
training  program,  and  what  modifications  are  called  for  by  the  analysis. 

This  Job  nid,  then,  addresses  the  problem  of  how  to  evaluate  lesson  plans 
for  adherence  to  sound  training  development  principles.  The  job  aid 
provides  guidance  that  enables  you  to  systematically  examine  lesson  plans 
and  Identify  those  areas  in  which  a  given  lesson  plan  is  deficient.  It 
allows  you  to  recognize  and  correct  many  training  and  testing  problems  in  a 
program  of  instruction  before  any  instruction  has  been  presented. 


SECTION  II 


t  x  i  i  wrx 

Well  before  the  training  program  ever  begins,  the  training  developers  will  have 
designed  lesson  plans  describing  the  training  program  and  how  it  will  be  con¬ 
ducted.  This  provides  an  opportunity  for  the  training  analyst  to  conduct  a 
preliminary  evaluation  of  the  training  program  by  examining  the  lesson  plans 
associated  with  that  program.  This  evaluation  may  be  performed  in  conjunction 
with  the  examination  of  the  lesson  plans  for  the  completion  of  Worksheet  1 
(see  Research  Product  81-16) . 

There  are  three  major  questions  that  can  be  investigated  at  this  point. 

1.  Are  the  goals  (training  objectives)  of  the  training  program  clearly  speci¬ 
fied?  If  the  goals  are  not  clearly  specified,  the  training  designer  does  not 
know  what  skills  and  knowledges  the  lesson  is  supposed  to  produce  in  the  sol¬ 
diers.  Learning  experiences  cannot  be  developed  because  the  designer  does  not 
know  exactly  what  skills  and  knowledges  the  soldiers  must  learn.  "What"  is 
taught  is  an  instructor  specific  decision.  There  is  no  guarantee  of  stability 
in  the  program  from  time  1  to  time  2  for  any  given  instructor  nor  from  in¬ 
structor  1  to  instructor  2  at  any  given  time. 

2.  Does  the  training  design  (lesson  plans)  specify  a  sequence  of  events  that 
have  been  shown  to  lead  to  goal  attainment?  The  training  design  (lesson  plan) 
must  specify  the  nature  and  sequence  of  the  learning  events.  These  events  must 
provide  the  soldiers  with  the  skills  and  knowledges  they  need  to  perform  the 
TASKS  in  the  training  objectives  to  the  required  STANDARD  under  the  training 
CONDITIONS.  The  lesson  plan  should  have  been  piloted  against  typical  input  to 
ensure  that  the  events  do,  in  fact,  achieve  the  training  objective.  If  the 
first  try  did  not  work,  the  lesson  plan  should  have  been  modified  and  piloted 
again.  This  process  needs  to  be  repeated  until  it  has  been  demonstrated  that 
the  training  design  does  work.  If  this  has  not  been  done,  then  there  is  no 
guarantee  that  the  training  program  will  achieve  the  training  objectives  even 
if  it  is  faithfully  Implemented.  It  probably  will  not  work  which  will  cause 
instructors  to  abandon  the  lesson  plan  and  give  instructor  specific  training. 

3.  Do  the  lesson  plans  provide  for  testing  the  soldiers  to  determine  if  they 
can  meet  the  goals  of  the  training  program  as  specified  in  the  training  objec¬ 
tives?  To  ensure  that  the  goals  of  the  training  program  are  met,  the  training 
plan  should  include  a  test  that  measures  each  soldier's  ability  to  perform  as 
described  in  the  training  objectives.  The  lesson  plan  should  include  instruc¬ 
tions  to  the  examiner  on  administering  the  test,  test  instructions  to  be  read 
to  the  soldier,  score  sheets  for  each  task  to  be  tested,  and  scoring  instruc¬ 
tions  for  the  examiner  including  test  conditions  and  standards.  The  lesson 
plans  should  also  provide  procedures  to  be  followed  when  a  soldier  receives  a 
NO-GO  on  the  test. 

If  training  managers  do  not  know  what  it  is  they  are  trying  to  do  (training 
objectives  not  clear)  and  they  do  not  know  if  what  they  are  planning  to  do 
about  it  (training  program  -  lesson  plans)  will  get  them  to  where  they  want  to 
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be  (soldier  performance  to  standard  on  the  objectives),  then  there  is  not  much 
sense  in  spending  resources  to  observe  training  and  testing  to  see  if  the  pro¬ 
gram  has  worked.  Effort,  at  this  point,  needs  to  be  put  into  goal  clarifica¬ 
tion  and  training  design. 

This  job  aid  contains  twenty-one  questions  that  should  be  answered  affirma¬ 
tively  in  examining  a  lesson  plan.  If  any  question  is  not  answered  in  the  af¬ 
firmative,  action  is  needed  to  change  either  the  objectives,  training  plan,  or 
test,  whichever  is  appropriate.  The  narrative  following  each  question  dis¬ 
cusses  the  nature  of  the  problem.  The  changes  needed  are,  of  course,  those 
which  will  cause  the  questions  to  be  answered  "yes." 

It  is  not  the  intent  of  this  job  aid  to  make  you  an  educational  technologist. 
The  proper  action  for  you  to  take  when  you  find  "no"  answers  to  these  questions 
is  to  refer  the  problem  back  to  the  training  developer.  The  training  developer 
may  not  agree  with  you.  Remember  that  your  job  is  to  identify  problems  and 
recommend  solutions,  not  to  force  compliance  with  your  position.  Whatever 
"leverage"  you  have  will  come  from  the  logic  of  your  data  and  the  backing  you 
get  from  higher  headquarters. 
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SECTION  III 


EVALUATING  TRAINING  OBJECTIVES 


1.  Does  the  training  objective  specify  what  the  soldier  must  do  after  Laving 
been  trained?  ” 


A  well  written  training  objective  is  the  heart  of  a  training  program  (see 
Section  II,  Research  Product  81-17).  It  tells  the  instructor,  soldier,  and 
the  training  manager  three  very  important  things:  what  it  is  that  the  soldier 
will  be  able  to  do  after  training  that  he  or  she  could  not  do  before  training 
(the  TASK),  the  conditions  under  which  the  soldier  will  be  able  to  do  it  in¬ 
cluding  what  the  soldier  will  have  available  to  help  with  the  performance  (the 
CONDITION),  and  how  well  the  soldier  must  perform  to  get  a  GO  (the  STANDARD). 

The  focus,  in  the  TASK  statement,  is  on  the  word  "DO."  Action  verbs  are  re¬ 
quired  to  describe  what  soldiers  do.  Soldiers  can  repair  things  when  they  are 
broken,  they  can  replace  things  when  they  do  not  work  any  more,  they  can  dis¬ 
assemble  things  that  are  already  put  together,  they  can  assemble  things  that 
have  been  taken  apart,  they  can  name  things  when  they  are  pointed  to,  they  can 
describe  what  they  would  do  when  something  goes  wrong.  It  is  best  fo  avoid 
more  general  terms  like  hit  targets,  be  oriented,  to  become  faniliar  with,  and 
take  appropriate  action,  because  they  are  not  specific  enough  to  adequately 
describe  the  task  to  be  accomplished. 

The  TASK  statement  provides  the  basis  for  the  training  design;  what  it  is  the 
soldiers  will  "DO"  in  training  that  will  build  the  skills  necessary  to  perform 
the  TASK.  If  the  TASK  statement  does  not  clearly  specify  what  it  is  that  the 
soldier  must  "DO"  after  training,  there  is  no  way  for  training  to  be  designed 
since  there  is  no  goal  to  reach.  Instructors  are  left  to  define  the  TASK  for 
themselves,  as  best  they  can  out  of  their  experience.  As  a  consequence,  train¬ 
ing  may  become  highly  instructor  dependent  with  each  instructor  conducting  his 
or  her  own  training  program.  "What"  gets  trained  varies  from  instructor  to 
instructor.  There  is  no  stable  training  program.  In  the  end,  what  gets  evalu¬ 
ated  is  a  collection  of  divergent  training  programs.  As  this  collection  shifts 
with  changes  in  instructor  personnel,  the  results  shift.  The  specific  profi¬ 
ciencies  you  get  today  may  not  be  the  same  proficiencies  you  get  next  time. 

This  general  argument  does  not  just  apply  to  the  statement  of  the  task.  It 
also  applies  to  the  specification  of  learning  activities  in  the  lesson  plan. 


The  point  here  is  that  before  you  can  evaluate  anything  you  have  to  be  able  to 
describe  it  and  be  reasonably  sure  that  "it"  will  remain  stable  during  the 
evaluation  period. 


2.  Is  it  clearly  a  training  objective  (and  not  a  job  requirement)? 


Training  objectives  are  derived  from  job  requirements  during  the  design  and 
development  of  a  training  program.  The  two  are  usually  different  since  soldiers 
are  usually  less  proficient  in  training,  are  performing  under  different  condi¬ 
tions,  and  often  to  different  standards  than  would  be  present  on  the  job.  You 
will  have  to  look  at  each  training  objective  closely  to  make  sure  that  it 
specifies  a  TASK,  CONDITION,  and  STANDARD  that  are  obtainable  in  the  training 
It  is  important  that  training  objectives  match  job  requirements 
but  within  the  constraints  of  the  military  training 
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environment, 
as  closely  as  possible,} 
situation. 


Sometimes  these  differences  are  small  but  they  are  still  important.  For  exam¬ 
ple,  if  the  TASK  requires  the  soldier  to  decontaminate  a  piece  of  equipment 
following  a  chemical  or  biological  attack  but  the  "real"  conditions  in  the 
training  environment  do  not  permit  the  soldier  to  be  subjected  to  such  a  haz¬ 
ard,  the  TASK  is  a  job  requirement  rather  than  a  training  objective.  The  dif¬ 
ferences  between  the  two  in  terms  of  instructor  and  soldier  behavior  are  very 
large.  In  the  presence  of  a  hazardous  agent  there  is  a  much  greater  sense  of 
urgency,  greater  stress,  greater  attention  to  soldier  performance  during  train¬ 
ing,  and  a  much  more  rigid  adherence  to  STANDARDS.  When  the  hazardous  agent 
is  simulated,  instructors  will  usually  let  slightly  substandard  performance, 
or  ^ven  a  verbal  description  satisfy  the  requirement.  The  slightly  substandard 
per  ormance  and  verbal  description  will  probably  not  be  clearly  defined  so,  as 
in  #1,  above,  what  is  learned  and  how  well  it  is  learned  will  vary,  from  in¬ 
structor  to  instructor. 

3.  Does  the  training  objective  specify  the  CONDITIONS  under  which  the  soldier 
must  demonstrate  task  performance? 

The  CONDITIONS  statement  should  lay  out  in  detail  the  environmental  conditions 
under  which  the  soldier  will  be  able  to  do  the  TASK  and  the  tools,  job  aids, 
etc.,  that  he  or  sne  will  have  available  to  help  with  TASK  performance  (see 
Section  II,  Research  Product  81-17). 

Again,  the  training  CONDITIONS  will  probably  differ  from  the  job  CONDITIONS. 

For  example,  soldiers  cannot  perform  tasks  under  NBC  conditions  in  training 
because  nuclear  and  biological  agents  cannot  be  used  in  a  training  environment. 
They  may  be  able  to  perform  under  certain  chemical  conditions,  however,  if  they 
are  clearly  specified.  If  soldiers  are  to  perform  under  conditions  of  dark¬ 
ness,  then  they  should  be  made  to  perform  when  it  is  dark.  If,  however, 
"darkness"  here  means  using  light  attenuating  goggles,  then  the  CONDITION 
should  specify  light  attenuating  goggles. 

If  TASK  performance  requires  the  use  of  a  tank,  or  a  rifle,  or  a  wrench,  or  a 
manual,  or  a  job  aid,  then  that  should  be  specified  and  provided  to  the  soldier 
when  he  or  she  is  required  to  perform  the  TASK. 

4.  Does  the  training  objective  specify  the  STANDARDS  to  which  soldiers  must 
perform? 

The  STANDARD  lays  out  in  detail  how  well  the  soldier  must  do  the  TASK  to  get  a 
GO  (see  Section  II,  Research  Product  81-17).  The  STANDARD  does  not  list  what 
the  soldier  does;  it  lists  the  criteria  for  knowing  when  it  has  been  done  right. 
It  tells  the  soldier  and  an  outside  observer  when  acceptable  performance  has 
been  reached. 

Standards  are  usually  written  in  terms  of  time  and/or  accuracy.  The  standard 
for  repairing  something  is  usually  expressed  in  terms  of  how  long  it  takes  and 
whether  it  works  as  it  should  when  repaired.  If  a  soldier  has  to  replace  some¬ 
thing,  he  or  she  usually  has  a  limited  amount  of  time  to  do  it  in  and  all  the 
connections  have  to  be  complete  and  accurate.  Disassembly  usually  involves 
time  and  all  the  right  pieces  being  taken  out,  perhaps  in  some  specified  se¬ 
quence.  The  sequence,  however,  is  stated  in  the  TASK,  not  the  STANDARD.  When 


something  is  assembled,  all  the  right  pieces  are  usually  b-’ck  in  and  it  works. 
In  all  of  these,  the  steps  to  be  performed  and  the  performance  sequence  .in- 
described  in  the  TASK.  The  STANDARD  tells  the  instructor  and  soldier  how  lone, 
he  or  she  has  to  do  it  in  and  refers  to  the  TASK  for  the  correct  performance 
steps  and  sequence. 

The  STANDARD  should  be  written  so  as  to  provide  a  GO/NC  GO  criterion  for 
testing. 

5.  Are  the  STANDARDS  clearly  spelled  out  so  that  the  soldier,  the  instructor 
the  examiner,  and  a  training  evaluator  can  tell  the  difference  between  perform¬ 
ance  at  or  above  standard  from  performance  that  is  below  standard? 

Standards  should  be  written  so  that  they  are  interpreted  by  everyone  in  the 
same  way.  For  this  to  occur,  standards  must  be  stated  in  unambiguous  language. 
Subjective  judgments  by  the  examiner  generally  do  not  constitute  adequate 
standards  and  are  only  slightly  better  than  no  standards  at  all.  Standards 
should  be  objective  measures  of  performance,  stated  in  terms  of  speed  and/or 
accuracy.  Stating  objective  standards  unambiguously  in  terms  of  speed  and/or 
accuracy  decreases  the  chances  that  the  standards  will  be  applied  differently 
by  different  examiners,  and  increases  the  chances  that  the  standards  will  be 
understood  by  the  soldiers,  the  instructor,  the  examiner,  and  the  evaluator. 

6.  Are  the  STANDARDS  stated  so  that  acceptable  performance  is  not  a  judgment 
call  on  the  part  of  an  instructor  (or  examiner)? 


Some  will  argue  that  standards  are  legitimately  judgment  calls  in  certain 
training  areas  (strategy,  tactics,  leadership,  etc.).  Do  not  yield  to  that 
argument  if  what  you  are  evaluating  is  training  (developing  specific  skills 
and  knowledges  in  soldiers)  rathfr*  than  education  'teaching  general  principles 
•/hich  the  soldier  may,  or  may  not,  apply  later).  If  everyone  agrees  that  the 
STANDARD  is  a  judgment  call,  and  the  individual  rules  for  making  the  judgment 
are  the  same,  they  can  be  written  down  and  called  a  STANDARD.  If  the  individual 
rules  differ,  then  there  is  no  consistency  (or  standardization)  in  the  program. 


SECTION  IV 

EVALUATING  LESSON  PLANS 


7.  Does  the  lesson  plan  call  for  the  dissemination  of  the  required  enabling 
knowledge  In  either  a  lecture  or  a  hand-out? 

Terms,  concepts,  or  other  subject  matter  that  are  included  in  a  lesson  to  en¬ 
hance  the  soldier's  understanding  of  the  tasks  to  be  learned  is  referred  to  as 
"enabling  knowledge."  The  lesson  plan  should  specify  the  enabling  knowledge 
needed  by  the  soldiers  for  task  performance.  It  should  specify,  as  a  first 
event  in  any  sequence  of  training  events,  a  discussion  or  lecture  where  the 
instructor  establishes  the  terms  and  concepts  needed  for  him  or  her  and  the 
soldiers  to  talk  about  the  task. 

The  lesson  plan  should  specify  what  enabling  knowledge  is  to  be  presented  and 
prescribe  procedures  (a  set  of  rules,  a  formula  to  follow,  etc.)  for  presenting 
it  since  the  soldiers  must  readily  understand  the  material. and  be  able  to  con¬ 
verse  easily  about  the  task  using  correct  terms  and  concepts.  The  lesson  plan 
should  specify  that  training  must  not  progress  to  a  practice  event  until  the 
instructor  is  sure  that  the  soldiers  understand  and  can  use  the  appropriate 
terms  and  concepts.  In  general,  the  lesson  should  introduce  terms  and  concepts 
when  they  are  needed  for  performance  rather  than  introduce  them  all  in  a  "terms 
and  concepts"  event  at  the  beginning  of  a  training  module. 

The  lesson  plan  should  specify  the  required  terms  and  concepts.  These  terms 
and  concepts  include  such  things  as  names  or  labels  for  things,  introduction 
of  new  equipment,  explanations  of  working  relationships  between  parts,  the 
establishment  of  required  readings  on  gauges  and  dials,  the  introduction  of 
rules  or  strategies,  the  explanation  of  underlying  concepts,  etc. 

8.  Does  the  lesson  plan  call  for  a  demonstration  of  the  task  (in  its  entirety) 

The  lesson  plan  should  call  for  a  demonstration  of  each  task,  including  the 
appropriate  steps  or  subtasks.  The  lesson  plan  should  provide  guidance  to  the 
instructor  in  conducting  the  demonstration;  it  should  prescribe  how  the  demon¬ 
stration  is  to  be  conducted.  Merely  listing  the  task  to  be  demonstrated,  and 
directing  the  instructor  to  demonstrate  the  necessary  maintenance,  assembly/ 
disassembly,  or  operating  procedures  is  not  sufficient. 

Typically  the  lesson  plan  should  call  for  demonstrations  to  be  conducted  by 
the  instructor  on  the  actual  equipment  (or  a  high  fidelity  mock-up)  in  full 
view  of  the  soldiers  being  trained.  This  provides  the  soldiers  with  a  live 
model  of  the  skilled  performance  required,  and  allows  the  soldiers  to  stop  the 
instructor  to  ask  questions.  Films,  slides,  or  other  pictorial  methods  are 
usually  less  effective  methods  of  demonstrating  tasks,  and  should  only  be  used 
when  time  and  resources  do  not  permit  a  live  demonstration  by  the  instructor. 


In  mosL  cases  the  lesson  plan  should  call  for  a  formal  demonstration  rather 
than  a  walk-through  or  talk-through.  A  walk-through  or  talk-through  may  be 
adequate  for  demonstrating  relatively  simple  tasks  where  skilled  performance 


is  not  required,  but  it  is  a  poor  substitute  for  a  formal  demonstration  in 
most  applications.  Extensive  supervised  practice  following  a  walk-through  or 
talk-through  can  sometimes  compensate  for  the  lack  of  a  formal  demonstration, 
but  in  general  the  lesson  plan  should  provide  for  a  formal  demonstration. 

9.  If  there  are  subtasks  in  the  lesson,  are  demonstrations  required  for  each 
subtask  and  an  integrated  demonstration  required  after  subtask  training? 

A  subtask  is  a  part  of  a  larger  task  that  can  be  performed  independent  of  the 
larger  task.  If  there  are  subtasks  in  the  lesson,  the  lesson  plan  should  call 
for  the  subtasks  to  be  demonstrated  either  as  part  of  the  overall  task  or  as 
separate  subtasks.  When  a  task  consists  of  a  number  of  related  subtasks  that 
are  part  of  a  larger  overall  task,  the  lesson  plan  should  call  for  a  demonstra¬ 
tion  of  each  followed  by  a  demonstration  of  the  full  task  to  include  each  of 
the  subtasks  performed  sequentially.  The  lesson  plan  should  specify  the  demon¬ 
stration  sequence  in  considerable  detail  and  show  how  to  schedule  demonstrations 
and  practice  together. 

10.  Does  the  lesson  plan  call  for  supervised  hands-on  practice? 

The  lesson  plan  should  call  for  practice  of  the  task  where  practice  is  defined 
as  hands-on  performance  of  the  task,  on  the  equipment  listed  in  the  training 
objective,  supervised  by  an  instructor  or  assistant  instructor,  during  train¬ 
ing.  Practice  events  are  essential  to  skill  development.  No  matter  how  good 
lectures,  demonstrations,  "talk-throughs,"  or  "walk-throughs"  are,  soldiers 
will  not  learn  to  perform  tasks  unless  they  practice  them  (see  Section  III, 
Research  Product  81-17) . 

11.  Does  the  lesson  plan  require  that  each  subtask  and  task  be  practiced? 

Both  subtask  and  task  practice  should  be  specified  in  the  lesson  plan.  Subtask 
practice  is  practice  on  a  part-task,  or  on  a  step  in  a  long  sequence,  or  on  a 
particular  set  of  actions  that  are  part  of  a  task.  Subtask  practice  is  neces¬ 
sary  when  a  task  is  composed  of  many  subtasks  (where  the  sequence  of  actions 
is  too  long  and  involved  for  the  soldier  to  master  in  one  practice  session). 

It  is  more  efficient  to  have  the  soldier  master  the  sequence  by  groups  of  sub¬ 
tasks.  After  several  subtasks  have  been  performed  to  standard,  then  the  sol¬ 
dier  can  begin  to  put  them  together.  The  practice  event  where  the  soldier 
attempts  to  put  the  subtasks  together,  to  practice  the  entire  task  under  the 
training  condition  to  the  training  standard,  is  called  whole-task  practice. 

12.  Does  the  lesson  plan  specify  that  practice  on  each  subtask/ task  will  be 
performed  by  each  soldier  to  a  specified  standard  (or  to  the  standard  specified 
in  the  training  objective)? 

The  lesson  plan  should  require  each  soldier  to  practice  each  task  and  subtask 
until  he  or  she  meets  the  standard  listed  in  the  training  objective  at  least 
once.  No  soldier  should  be  allowed  to  progress  in  training  without  having 
satisfied  the  training  objective.  Novel  or  difficult  tasks  may  require  more 
than  one  demonstration  of  proficiency  by  each  soldier. 
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13.  Does  the  lesson  plan  tell  AIs  what  to  look  for  during  practice  and  how  to 
correct  faulty  performance?  ~ 

Progress  should  be  made  during  practice.  As  obvious  as  this  point  seems.  It 
is  often  ignored.  Progress  is  crucial,  for  if  it  does  not  occur,  the  soldier 
is  practicing  at  least  some  incorrect  performance.  Furthermore,  because  prog¬ 
ress  should  occur,  failure  to  make  progress  is  a  good  indication  that  something 
in  the  training  needs  correcting.  Judgments  as  to  whether  progress  is  occur¬ 
ring  must  be  made  by  assistant  instructors  who  are  monitoring  practice.  This 
means  that  the  lesson  plan  should  tell  the  assistant  instructors  what  perform¬ 
ance  problems  are  likely,  how  to  identify  them,  how  to  provide  feedback,  and 
what  remedial,  exercises  to  use  to  correct  faulty  performance.  For  example, 
the  lesson  plan  should  specify  those  aspects  of  the  tasks  that  are  likely  to 
cause  problems  for  the  soldiers,  list  common  soldier  errors  on  the  tasks  being 
practiced,  and  direct  the  instructor  to  make  feedback  regarding  soldier  errors 
immediate,  specific  to  the  actions  performed,  and  free  of  ridicule. 

The  lesson  plan  must  also  include  a  requirement  for  enough  assistant  instruc¬ 
tors  to  insure  that  no  practice  is  conducted  without  an  assistant  instructor 
present.  Instructors  must  be  available  to  provide  demonstrations  when  neces¬ 
sary,  point  out  cues  and  critical  discriminations,  provide  feedback,  and  keep 
the  soldier  from  practicing  incorrect  actions.  Soldiers  learn  what  they  prac¬ 
tice  (see  Section  III,  Research  Product  81-17).  They  learn  incorrect  actions 
just  as  readily  as  correct  actions.  Until  a  soldier  is  able  to  perform  a  task 
without  error,  or  can  recognize  and  correct  his/her  own  errors,  the  soldier 
should  practice  under  the  direct  supervision  of  an  instructor. 
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SECTION  V 


KVAI.DATINC  TKSTINi:  I’l.ANS  AND  INSTRUMENT:: 


14.  Do  the  test  Items  derive  directly  from  the  training  objectives? 

The  TASK  statements  in  the  training  objectives  should  determine  what  is  trained 
and  what  is  tested.  Consequently,  tasks  in  the  training  objectives,  tasks 
trained  and  tasks  tested  should  be  the  same.  If  the  training  objective  says 
that  the  soldier  should  "perform"  emergency  procedures,  then  the  test  require¬ 
ment  should  not  specify  that  the  soldier  "simulate"  the  emergency  procedures, 
or  if  the  objective  states  that  the  soldier  "perform"  troubleshooting  in 
response  to  a  specified  malfunction  the  test  should  not  ask  that  the  soldiers 
"explain"  the  troubleshooting  procedures  for  that  malfunction. 

Tasks  that  are  Important  enough  to  be  included  in  the  training  objectives 
should  be  included  on  the  test.  When^ tasks  are  omitted  from  the  test,  the 
training  analyst  has  less  basis  for  evaluating  the  effectiveness  of  the  train¬ 
ing  for  those  tasks.  Furthermore,  tasks  that  are  not  tested  are  often  de- 
emphasized  during  training,  thereby  reducing  training  effectiveness  for  those 
tasks.  Similarly,  soldiers  should  not  be  tested  on  tasks  that  are  not  included 
in  the  statement  of  the  training  objective.  If  this  occurs  soldiers  may  be 
tested  on  a  task  for  which  they  were  not  trained. 

15.  Does  the  test  require  the  soldier  to  perform  the  steps  specified  in  the 
training  objective? 

Following  naturally  from  #14,  above,  the  test  should  require  the  performance 
of  the  steps  or  subtasks  in  the  training  objective.  If  a  step  or  subtask  is 
important  enough  to  be  singled  Out  and  identified  in  the  training  objective, 
it  is  important  enough  to  test. 

Critical  decision  and/or  difficult  actions  identified  in  the  lesson  plan  should 
be  part  of  the  test.  These  are  often  not  included  on  tests  because  1)  they 
are  hard  to  train  to  standard  and  instructors  often  doubt  the  ability  of  sol¬ 
diers  to  demonstrate  proficiency,  and  2)  they  are  hard  to  test.  They  should 
be  included  on  the  test,  however,  exactly  for  these  reasons  and  because  the 
test  cannot  measure  program  success  and  soldier  mastery  of  the  training  objec¬ 
tives  if  the  critical  decisions  and/or  difficult  actions  are  omitted. 

All  the  critical  discriminations  included  in  the  training  program  should  also 
be  included  in  the  test.  Remember  that  there  are  four  parts  to  performing  any 
task;  1)  knowing  when  action  is/is  not  required,  2)  knowing  which  actions  (of 
several)  are  required  in  tils  case,  3)  performing  the  required  actions  correctly 
and  to  standard,  and  4)  recognizing  that  actions  being  performed  are  correct 
and  to  standard.  The  basesi  for  deciding  when  action  is/is  not  required  or 
which  actions  (of  several)  to  take  in  this  case  are  the  critical  discrimina¬ 
tions.  They  are  often  not  included  on  tests  because  they  are  hard  to  set  up. 


16.  Are  the  conditions  specified  for  the  test  the  same  as  those  specified  for 
the  training  in  the  training  objective? 

The  CONDITIONS  statement  in  the  training  objectives  should  determine  the  condi¬ 
tions  for  both  training  and  testing.  Consequently  the  conditions  for  training 
and  testing  should  be  the  same.  All  conditions  specified  in  the  training  ob¬ 
jectives  should  be  included  in  the  test,  and  no  new  conditions  should  be  added 
for  testing  purposes  that  were  not  listed  in  the  training  objectives.  Make 
sure  that  the  conditions  listed  in  the  training  objectives  and  on  the  test  are 
exactly  the  same.  For  example,  if  the  training  objective  requires  the  task  to 
be  performed  under  conditions  of  reduced  visibility  (which  could  include  smoke, 
fog,  darkness),  a  test  condition  requiring  performance  only  under  conditions  of 
darkness  is  not  equivalent . 

17.  Are  the  standards  specified  for  the  test  the  same  as  those  specified  in 
the  training  objective? 

The  STANDARDS  listed  in  the  training  objectives  should  determine  the  standards 
to  be  used  during  the  latter  stages  of  practice  and  on  the  test.  Test  stan¬ 
dards  should  be  specified  in  the  instructions  to  the  examiner  or  on  the  test 
scoresheets.  If  the  training  objective  requires  that  a  task  be  completed  in  a 
given  amount  of  time,  then  the  same  amount  of  time  to  complete  the  task  should 
be  allotted  on  the  test.  Similarly,  if  the  objective  specifies  that  the  sol¬ 
dier's  performance  must  be  within  certain  accuracy  limits,  then  the  test 
should  require  the  same  degree  of  accuracy  from  the  soldier.  If  test  standards 
differ  from  the  standards  listed  in  the  training  objective,  you  should  try  to 
convince  the  training  developer  to  make  the  standards  consistent.  Otherwise, 
you  may  have  no  basis  for  determining  if  the  soldiers  can  perform  to  the  stan¬ 
dard  listed  in  the  training  objective. 

18.  Are  the  instructions  for  administering  the  test  such  as  to  ensure  stan¬ 
dardization  across  examiners? 

The  instructions  for  administering  the  test  should  be  written  so  as  to  ensure 
standardization  of  test  administration  procedures  across  examiners.  The  lesson 
plan  should  outline  precisely  how  the  test  is  to  be  administered.  Included 
should  be  the  placement  of  personnel  and  equipment  and  test  administration 
procedures  that  specify  the  testing  sequence  and  guide  the  evaluator  in  testing 
the  soldiers  on  each  of  the  tasks  to  be  performed.  Such  guidance  for  the 
evaluator  is  needed  in  order  to  ensure  that  each  soldier  is  tested  in  the  same 
way. 

19.  Does  the  lesson  plan  include  instructions  for  the  examiner  to  read  to  the 
soldier  on  how  the  test  will  be  conducted  and  scored? 

The  lesson  plan  should  include  instructions  to  the  soldier  explaining  how  the 
test  will  be  conducted  and  scored.  These  instructions  should  be  clear  and 
complete  to  ensure  that  testers  do  not  have  to  add  instructions  of  their  own. 

(nst  met  Jons  should  fell  f  lie  soldier  what  he  or  she  will  have  to  do,  what 
••tie:;  f  hey  will  have  to  know  to  determine  when  to  start,  the  conditions  they 
will  have  to  perform  under,  the  equipment/ tools  they  will  have  and  can  use,  how 
much  time  they  will  have,  and  the  standards  for  successful  completion  of  the 
task. 
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20.  Are  scoring  instructions  specific  enough  (at  the  step/subtask  level)  to 
ensure  that  correct  performance  Is  observable  and  understandable  to  both  l  lie 
examiner  and  the  soldier? 

Instructions  for  scoring  the  test  should  specify  soldier  actions  that  can  be 
seen  or,  when  soldier  actions  cannot  be  seen,  the  results  of  those  actions 
evidenced  by  a  completed  sequence  or  product.  If  how  a  soldier  performs  a 
task  is  important,  then  the  action  sequence  should  be  scored.  If  only  the 
final  product  of  that  performance  is  important  and  how  the  soldier  gets  to 
that  final  product  is  irrelevant,  then  only  the  product  need  be  scored. 

Scoring  instructions  should  be  written  simply  so  that  soldiers,  as  well  as 
examiners,  can  easily  understand  them. 


21.  Does  the  test  plan  call  for  explaining  or  demonstrating  to  a  soldier  what 
he/she  did  wrong  during  the  test? 

Sometimes  it  is  not  at  all  clear  to  an  individual  why  he/ she  is  making  a  mis¬ 
take  in  some  procedure  (aiming  and  firing  a  rifle  is  a  good  example).  The 
soldier  may  think  that  he/she  is  doing  everything  correctly,  but  he/she  keeps 
failing  the  task.  Specific  correction  is  needed  in  such  situations,  and  some¬ 
times  a  demonstration  of  how  the  soldier  is  incorrectly  performing  the  task  is 
very  useful. 
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